-
Notifications
You must be signed in to change notification settings - Fork 303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add cross-region inference support for Bedrock models (#535) #536
base: v1
Are you sure you want to change the base?
Conversation
…les#535) This commit introduces the initial setup for supporting cross-region inference in the Bedrock Chat application. The changes include: - Added `is_region_supported_for_inference` function in `utils.py` to check if a region supports inference. - Modified `get_bedrock_client` function in `utils.py` to use cross-region inference if enabled and the region is supported. - Updated `get_model_id` function in `bedrock.py` to use the base model ID for cross-region inference if enabled, the region is supported, and the model is included in `CROSS_REGION_INFERENCE_MODELS`. If any of these conditions are not met, it falls back to using the local model ID and logs a warning. - Added `enableBedrockCrossRegionInference` option to `cdk.json` with the default value set to `false`. These changes lay the foundation for enabling cross-region inference in the Bedrock Chat application. The feature can be enabled or disabled using the `enableBedrockCrossRegionInference` configuration option in `cdk.json`.
…les#535) - Fixing '(' was never closed
…les#535) - Bedrock and utils reformated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chm10 Please review my comment. Please check whether your code works as intended before commit, and also note the condition and results you tested. Thank you!
@@ -40,9 +43,29 @@ def is_running_on_lambda(): | |||
return "AWS_EXECUTION_ENV" in os.environ | |||
|
|||
|
|||
def is_region_supported_for_inference(region: str) -> bool: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To emphasize for cross region inference, rename it like:
before: is_region_supported_for_inference
.
after: is_region_supported_for_cross_inference
@@ -40,9 +43,29 @@ def is_running_on_lambda(): | |||
return "AWS_EXECUTION_ENV" in os.environ | |||
|
|||
|
|||
def is_region_supported_for_inference(region: str) -> bool: | |||
supported_regions = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this variable is constant, so please replace with SUPPORTED_REGIONS
logger.warning( | ||
f"Cross-region inference is enabled, but the region {region} is not supported. Using default region." | ||
) | ||
return boto3.client("bedrock-runtime", region_name=REGION) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use region
i.e. BEDROCK_REGION
as default client. REGION
is used for services except for bedrock
@@ -257,26 +265,48 @@ def calculate_price( | |||
return input_price * input_tokens / 1000.0 + output_price * output_tokens / 1000.0 | |||
|
|||
|
|||
CROSS_REGION_INFERENCE_MODELS = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
move it inside get_model_id
since it's used only for the function
logger.warning( | ||
f"Cross-region inference is not available for model {model}. Using local model." | ||
) | ||
return f"{base_model_id}-local" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the postfix i.e. -local
work?
logger.info( | ||
f"Using cross-region inference for model {model} in region {BEDROCK_REGION}" | ||
) | ||
return base_model_id |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to add prefix e.g. eu
, us
?
This commit introduces the initial setup for supporting cross-region inference in the Bedrock Chat application.
Issue #, if available:
#535
Description of changes:
is_region_supported_for_inference
function inutils.py
to check if a region supports inference.get_bedrock_client
function inutils.py
to use cross-region inference if enabled and the region is supported.get_model_id
function inbedrock.py
to use the base model ID for cross-region inference if enabled, the region is supported, and the model is included inCROSS_REGION_INFERENCE_MODELS
. If any of these conditions are not met, it falls back to using the local model ID and logs a warning.enableBedrockCrossRegionInference
option tocdk.json
with the default value set tofalse
.These changes lay the foundation for enabling cross-region inference in the Bedrock Chat application. The feature can be enabled or disabled using the
enableBedrockCrossRegionInference
configuration option incdk.json
.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.